Conventionally, there is a game apparatus capable of taking an image of a face of each of players by imaging means and allowing the players to fight against each other using such images (see, for example, Japanese Laid-Open Patent Publication No. 2004-173960). This game apparatus reads an image of the player's face, extracts face feature point data of the read image of the face and calculates score data from the face feature point data using a specific calculation equation and a specific coefficient. The pieces of score data thus obtained regarding a plurality of players are compared with each other to determine which player wins.
However, the game apparatus described in Japanese Laid-Open Patent Publication No. 2004-173960 does not have a function of instructing the player to make a predetermined facial expression or a function of evaluating the facial expression made by the player in accordance with the instruction. Therefore, this game apparatus is not usable for training mimetic muscles.
As supporting tools for a training of mimetic muscles, books having drawings or photos representing model expressions are conventionally provided. For training mimetic muscles with such books, the user needs to evaluate himself/herself whether he/she can make a facial expression like the model while looking at a mirror and thus cannot obtain any objective evaluation.
Therefore, a feature of the present invention is to provide a computer-readable storage medium having stored thereon a training program allowing the user to train his/her mimetic muscles effectively and a training apparatus usable for such a training program.
The present invention has the following features. The reference numerals in parentheses in this section of the specification indicate the correspondence with the embodiments described later and the drawings for easier understanding of the present invention, and do not limit the present invention in any way.
A storage medium according to the present invention is a computer-readable storage medium having stored thereon a training program (50) for supporting a training of mimetic muscles. The training program causes a computer (21) to execute a selection step (S23), a face image obtaining step (S51), a face feature point detection step (S51), an evaluation step (S24, S26), and an evaluation result presentation step (S29). The selection step selects a given expression to be made by a user among a plurality of prepared given expressions. The face image obtaining step obtains a face image of the user by an imaging device (38) connected to the computer. The face feature point detection step detects a position of a face feature point corresponding to at least the given expression selected in the selection step, from the face image obtained in the face image obtaining step. The evaluation step evaluates the user's expression in the face image obtained in the face image obtaining step based on the position of the face feature point detected in the face feature point detection step, in accordance with an evaluation criterion corresponding to the given expression selected in the selection step among evaluation criteria (52) respectively preset for the plurality of given expressions. The evaluation result presentation step presents a result of the evaluation in the evaluation step to the user through an image or an audio signal.
The evaluation criteria may be preset such that a different face feature point is to be referred to in accordance with a different given expression; and in the evaluation step, the user's expression in the face image obtained in the face image obtaining step may be evaluated by referring to a position of a face feature point corresponding to the given expression selected in the selection step. Owing to this, the user's expression can be appropriately evaluated in accordance with the given expression.
In the face feature point detection step, positions of at least two face feature points corresponding to the given expression selected in the selection step may be detected from the face image obtained in the face image obtaining step; and in the evaluation step, the user's expression in the face image may be evaluated based on the positional relationship between the two face feature points detected in the face feature point detection step.
The evaluation step may include a step of calculating a distance between the two face feature points detected in the face feature point detection step; and a step of evaluating the user's expression in the face image obtained in the face image obtaining step based on the calculated distance. Owing to this, the user's expression can be evaluated from the viewpoint of how much the specific two face feature points are separated from each other.
Target distance data may be preset for each of the plurality of given expressions; and in the evaluation step, the user's expression in the face image obtained in the face image obtaining step may be evaluated by comparing the distance calculated in the step of calculating the distance and a distance represented by the target distance data corresponding to the given expression selected in the selection step among the target distance data preset for each of the plurality of given expressions.
In the evaluation step, the user's expression in the face image obtained in the face image obtaining step may be evaluated based on the positional relationship between the position of the face feature point detected in the face feature point detection step and a reference position of the face feature point stored on a storage device (24).
Target distance data may be preset for each of the plurality of given expressions; and the evaluation step may include a step of calculating a distance between the position of the face feature point detected in the face feature point detection step and a reference position of the face feature point stored on the storage device; and a step of evaluating the user's expression in the face image obtained in the face image obtaining step by comparing the calculated distance and a distance represented by the target distance data corresponding to the given expression selected in the selection step among the target distance data preset for each of the plurality of given expressions.
The training program may cause the computer to further execute a reference face image obtaining step (S51), a reference position detection step (S51), and a reference position storage step (S22). The reference face image obtaining step obtains a face image of the user in an expressionless state by the imaging device as a reference face image at least before the face image obtaining step. The reference position detection step detects a position of a preset face feature point in the face image obtained in the reference face image obtaining step. The reference position storage step stores, on the storage device, the position of the face feature point detected in the reference position detection step as a reference position of the face feature point. Owing to this, the reference position of the face feature point is set based on the face image of the user. Thus, the user's expression can be more appropriately evaluated.
In the reference position detection step, positions of a plurality of face feature points in the face image obtained in the reference face image obtaining step may be detected; in the reference position storage step, the positions of the plurality of face feature points may be stored on the storage device; in the face feature point detection step, the positions of the plurality of face feature points may be detected from the face image obtained in the face image obtaining step; either one of the plurality of face feature points may be designated as a face feature point to be used for evaluating the face image for each of the plurality of given expressions; and in the evaluation step, the user's expression in the face image obtained in the face image obtaining step may be evaluated by comparing the position of the face feature point corresponding to the given expression selected in the selection step among the plurality of face feature points detected in the face feature point detection step and the position of the face feature point corresponding to the given expression selected in the selection step among the plurality of face feature points stored in the reference position storage step, in accordance with the evaluation criterion corresponding to the given expression selected in the selection step.
The evaluation step may include a step of calculating a distance between the position of the face feature point detected in the face feature point detection step and a reference position of the face feature point stored on the storage device; and a step of evaluating the user's expression in the face image obtained in the face image obtaining step based on the calculated distance. Owing to this, the user's expression can be evaluated from the viewpoint of how much the face feature point in the user's face image has moved from the reference position.
In the evaluation step, a score of the face image obtained in the face image obtaining step may be calculated based on the position of the face feature point detected in the face feature point detection step in accordance with the evaluation criterion corresponding to the given expression selected in the selection step among the evaluation criteria respectively preset for the plurality of given expressions; and in the evaluation result presentation step, the score calculated in the evaluation step may be presented to the user through an image or an audio signal. Owing to this, the score of the user's expression can be calculated from the viewpoint of how much the face feature point in the user's face image has moved from the reference position. Here, the “score” may be any of “item score”, “score of the face image”, “zone score”, “score of the moving exercise” and “score of the still exercise” in the embodiments described later.
In the face image obtaining step, a plurality of face images taken at different times during a predetermined time period may be obtained; in the face feature point detection step, a position of a face feature point corresponding to at least the given expression selected in the selection step may be detected from each of the plurality of face images obtained in the face image obtaining step; and in the evaluation step, the user's expression in the plurality of face images obtained in the face image obtaining step may be evaluated based on the positions of the face feature points in the plurality of face images detected in the face feature point detection step. Owing to this, the user's expression can be evaluated from a more diversified range of viewpoints than the case where the user's expression is evaluated based on one frame image.
In the face image obtaining step, the plurality of face images may be each obtained at a cycle of a constant time period. Owing to this, the position of the face feature point in the middle of the change of the user's expression can be obtained.
The evaluation step may include a step of calculating a score of each of the plurality of face images obtained in the face image obtaining step in the predetermined time period based on the position of the face feature point in the respective face image detected in the face feature point detection step; and a step of evaluating the user's expression in the plurality of face images obtained in the face image obtaining step in the predetermined time period, using scores of the plurality of face images other than at least one score which is equal to or lower than a certain value or which is relatively low. Owing to this, the score of the face image can be varied in consideration of the time-wise change of the user's expression.
The evaluation step may include a step of calculating a score of each of the plurality of face images obtained in the face image obtaining step in the predetermined time period based on the position of the face feature point in the respective face image detected in the face feature point detection step; and a step of obtaining an average of the scores of the plurality of face images obtained in the face image obtaining step in the predetermined time period, and evaluating the user's expression in the face images using the average.
The evaluation step may include a first evaluation step of evaluating the user's expression in each of the plurality of face images obtained in the face image obtaining step based on the position of the face feature point in the respective face image detected in the face feature point detection step, such that the user's expression is evaluated higher as the face image is closer to the given expression selected in the selection step; and a second evaluation step of evaluating the user's expression in accordance with whether or not a face image taken later is evaluated higher, based on the evaluation result of the first evaluation step. Owing to this, it can be evaluated whether or not the user's expression is smoothly changed.
The evaluation step may include a step (S52) of calculating the score of each of the plurality of face images obtained in the face image obtaining step based on the position of the face feature point detected in the face feature point detection step; a step of correcting the score of at least one face image (for example, the fifth zone score in the embodiment described later) in accordance with whether or not a face image taken later is evaluated higher; and a step of evaluating the user's expression in accordance with at least the corrected score. Owing to this, the score of the face image can be varied in accordance with whether or not the user's expression is smoothly changed.
The training may be for smoothly changing an expressionless state to a given expression at a constant speed; and the evaluation step may include a step of calculating the score of each of the face images obtained in the face image obtaining step based on the position of the face feature point in the respective face image detected in the face feature point detection step, such that a face image closer to the given expression selected in the selection step is evaluated higher; and a step (S47) of evaluating the user's expression in accordance with whether or not the scores of the face images linearly increase in accordance with the time at which the face images are taken. Owing to this, it can be evaluated whether or not the user's expression is smoothly changed.
The evaluation step may include a step (S52) of calculating the score of each of the face images obtained in the face image obtaining step based on the position of the face feature point in the respective face image detected in the face feature point detection step; a step (S47) of correcting the score of at least one face image (for example, the fifth zone score in the embodiment described later) in accordance with whether or not the scores of the face images linearly increase in accordance with the time at which the face images are taken; and a step of evaluating the user's expression in accordance with at least the corrected score. Owing to this, the score of the face image can be varied in accordance with whether or not the user's expression is smoothly changed.
In the face image obtaining step, a plurality of face images taken at different times may be obtained; in the face feature point detection step, a position of a face feature point corresponding to at least the given expression selected in the selection step may be detected from each of the plurality of face images obtained in the face image obtaining step; and the evaluation step may include a step (S54) of calculating a score of each of the plurality of face images obtained in the face image obtaining step based on the position of the face feature point in the respective face image detected in the face feature point detection step; a step (S57) of calculating at least one of a maximum value, a minimum value, an average value and a partial average value of the scores of the plurality of face images; and a step of evaluating the user's expression in accordance with the at least one of the maximum value, the minimum value, the average value and the partial average value. Owing to this, the user's expression can be evaluated with high precision without relying on the detection precision of the face feature point.
The evaluation step may include a step (S54) of calculating a score of each of the plurality of face images obtained in the face image obtaining step based on the position of the face feature point in the respective face image detected in the face feature point detection step; a step (S57) of calculating an average of the scores of a part of the plurality of face images having a relatively high score as a partial average value of the plurality of face images; and a step of evaluating the user's expression in accordance with the partial average value. Owing to this, the user's expression can be evaluated with high precision without relying on the detection precision of the face feature point.
In the face image obtaining step, a plurality of face images taken at different times during a predetermined time period (for example, in the moving exercise period in the embodiment described later) may be obtained; in the face feature point detection step, a position of a face feature point corresponding to at least the given expression selected in the selection step may be detected from each of the plurality of face images obtained in the face image obtaining step; and the evaluation step may include a step (S54) of calculating a score of each of a plurality of face images obtained in each of a plurality of zones (for example, first through fifth zones in the embodiment described later) included in the predetermined time period, based on the position of the face feature point in the respective face image detected in the face feature point detection step; a step (S57) of calculating a zone score of each zone based on the scores of the plurality of face images obtained in the respective zone; and a step of evaluating the user's expression in accordance with the zone scores of the plurality of zones (FIG. 22). Owing to this, the user's expression can be evaluated with high precision without relying on the detection precision of the face feature point.
In the face feature point detection step, positions of at least a first face feature point and a second face feature point may be detected from the face image obtained in the face image obtaining step; and the evaluation step may include a step of increasing (S74) a score of the face image obtained in the face image obtaining step based on the positional relationship between the position of the first face feature point detected in the face feature point detection step and a reference position of the first face feature point stored on the storage device; and a step (S74) of decreasing the score of the face image obtained in the face image obtaining step based on the positional relationship between the position of the second face feature point detected in the face feature point detection step and a reference position of the second face feature point stored on the storage device. Owing to this, the user's expression may be appropriately evaluated in an exercise of, for example, moving one face feature point without moving the other face feature points.
In the face feature point detection step, positions of at least a first face feature point and a second face feature point may be detected from the face image obtained in the face image obtaining step; and the evaluation step may include a step (S74) of increasing a score of the face image obtained in the face image obtaining step based on the positional relationship between the position of the first face feature point detected in the face feature point detection step and a reference position of the first face feature point stored on the storage device; a step (S74) of increasing the score of the face image obtained in the face image obtaining step based on the positional relationship between the position of the second face feature point detected in the face feature point detection step and a reference position of the second face feature point stored on the storage device; and a step (S74) of varying the score of the face image obtained in the face image obtaining step based on the positional relationship between the position of the first face feature point and the position of the second face feature point both detected in the face feature point detection step. Owing to this, the user's expression may be appropriately evaluated in an exercise of, for example, moving specific two face feature points in a well balanced state.
The training program may cause the computer to further execute a face coordinate system setting step (S34 through S36) of, after the face feature point detection step, setting an origin and a coordinate axis of a face coordinate system based on positions of at least two predetermined face feature points among a plurality of face feature points detected in the face feature point detection step; and a step (S37 through S38) of converting a coordinate set representing the position of each of the face feature points detected in the face feature point detection step to a coordinate set of the face coordinate system set in the face coordinate system setting step.
In the face coordinate system setting step, a direction of the coordinate axis (X axis, Y axis) of the face coordinate system may be set based on a direction of a straight line connecting at least two predetermined face feature points (P13, P22) among the plurality of face feature points detected in the face feature point detection step. Owing to this, even when the user's face is rotated around the optical axis of the lens of the imaging device, the coordinate set of each face feature point does not change accordingly. Therefore, the user's expression can be evaluated accurately.
In the face coordinate system setting step, a scale (the magnitude of X axis unit vector, the magnitude of Y axis unit vector) of the coordinate axis (X axis, Y axis) of the face coordinate system may be set based on a distance between at least two predetermined face feature points (P13, P22) among the plurality of face feature points detected in the face feature point detection step. Thus, even when the distance between the imaging device and the user's face is changed, the coordinate set of each face feature point does not change accordingly. Therefore, the user's expression can be evaluated accurately.
The training program may cause the computer to further execute a step of displaying a model image (40) corresponding to the given expression selected in the selection step on a screen (11). Owing to this, the user can perform an exercise of the mimetic muscles while looking at the model image, and thus can perform the exercise effectively.
The model image may be an animation image (FIG. 6). Owing to this, when performing an exercise of changing the expression slowly, for example, the user can perform an exercise of the mimetic muscles while looking at the model image represented by the animation image and thus can perform the exercise effectively.
The training program may cause the computer to further execute a step of displaying a face image of the user taken by the imaging device on a screen (12). Owing to this, the user can perform an exercise of the mimetic muscles while looking at his/her actual expression, and thus can perform the exercise effectively.
The training program may cause the computer to further execute a step of displaying a model image corresponding to the given expression selected in the selection step and a face image of the user taken by the imaging device on an identical screen or on different screens (11, 12) simultaneously (FIG. 7). Owing to this, the user can perform an exercise of the mimetic muscles while comparing the model image and his/her actual expression, and thus can perform the exercise effectively.
A training apparatus (10) according to the present invention is for supporting a training of mimetic muscles. The training apparatus comprises selection means (21), face image obtaining means (21), face feature point detection means (21), evaluation means (21), and evaluation result presentation means (21). The selection means selects a given expression to be made by a user among a plurality of prepared given expressions. The face image obtaining means obtains a face image of the user by an imaging device (38). The face feature point detection means detects a position of a face feature point corresponding to at least the given expression selected by the selection means, from the face image obtained by the face image obtaining means. The evaluation means evaluates the user's expression in the face image obtained by the face image obtaining means based on the position of the face feature point detected by the face feature point detection means, in accordance with an evaluation criterion corresponding to the given expression selected by the selection means among evaluation criteria (52) respectively preset for the plurality of given expressions. The evaluation result presentation means presents a result of the evaluation by the evaluation means to the user through an image or an audio signal.
According to the present invention, the user can obtain an objective evaluation result when training the mimetic muscles and thus can perform the training effectively.
These and other objects, features, aspects and advantages of the present invention will become more apparent from the following detailed description of the present invention when taken in conjunction with the accompanying drawings.